Image processing device and image display device

ABSTRACT

The input device and the input system include a hand detection means which detects the position of the hand, a body part detection means which detects positions of user&#39;s body parts such as the face, for example, a relative position calculation means which calculates a relative position of the hand with respect to body parts from hand position information being a detection result of the hand detection means and body part position information being a detection result of the body part detection means, and a gesture recognition means which recognizes a hand gesture on the basis of a change in the hand position information being the detection result of the hand detection means, wherein when a hand gesture is recognized, the operation of the input device with respect to the user&#39;s gesture is changed according to a relative position of the hand with respect to body parts.

INCORPORATION BY REFERENCE

This application relates to and claims by priority from Japanese PatentApplication No. 2010-289122 filed on Dec. 27, 2010, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

(1) Field of the Invention

The present invention relates to the operation of electronic devices.

(2) Description of the Related Art

An invention disclosed by a Japanese Patent Application Laid-Open No.2003-233452 (hereinafter, referred to as Patent Document 1) is known asa prior art in the field of the present invention. The Patent Document 1describes that an object of the prior art is “to provide a user-friendlygesture command input device which allows a user to input a command byusing a user's favorite gesture irrespective of applications or gesturecommands arbitrarily registered by other users and provides an inputmeans which recognizes a specific user performing a gesture input andallows multiple users to perform command inputs by using gesturesspecific to the user” (see [0013] of Patent Document 1), and alsodescribes that the prior art provides, as means for solving the problem,“a gesture command input device which converts a gesture input performedwith respect to an input means to a command, the gesture input beingperformed by using a user's body part or a dedicated command medium, thegesture command input device including: a storage means in whichuser-specific gesture commands are pre-registered per user, a userrecognition means which recognizes the user who performs the gestureinput with respect to the input means, and a control means which, whenthe user recognition means recognizes the user who performs the gestureinput with respect to the input means, converts a gesture input by theuser through the input means to a command based on a user-specificgesture command registered in the storage means in association with therecognized user” (see [0025] of Patent Document 1).

SUMMARY OF THE INVENTION

The device according to the technical idea of the Patent Document 1,however, is not easy for users to operate, since pre-registered gesturesneed to be stored in electronic units.

In order to solve the above problem, the present invention adopts, forexample, the configurations claimed in the appended Claims.

The present invention provides a plurality of means to solve the aboveproblem. One example is an image processing device configured to outputthe image signal, the image processing device including a cameraconfigured to capture the image, and a control unit configured tocontrol the image processing device based on the image captured by thecamera, wherein the control unit detects user's multiple body parts fromthe image captured by the camera, selects a type of the operation withrespect to the image processing device based on positions of thedetected multiple body parts, and when recognizing a user'spredetermined gesture from the image captured by the camera, executes anoperation of the selected type.

The above-mentioned means enables efficient operations of the electronicunits by the gesture and improves user's usability of the electronicunits.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, objects and advantages of the presentinvention will become more apparent from the following description whentaken in conjunction with the accompanying drawings wherein:

FIG. 1 is a schematic view showing an input device according to a firstembodiment,

FIG. 2 is a block diagram showing a configuration of the input deviceaccording to the first embodiment,

FIG. 3 is a schematic view showing the positions of the user's hand andface when the user operates the input device according to the firstembodiment,

FIGS. 4A and B are schematic views illustrating the position of theuser's hand and corresponding display screens of the input deviceaccording to the first embodiment,

FIG. 5 is a flowchart illustrating the operations of the input deviceaccording to the first embodiment,

FIG. 6 is a schematic view showing an input device according to a secondembodiment,

FIG. 7 is a schematic view of the input device according to the secondembodiment, showing regions of the user's gesture operation,

FIGS. 8A and B are schematic views of the input device according to thesecond embodiment, showing motions of the user's hand and correspondingdisplay screens,

FIG. 9 is a schematic view of an input device according to a thirdembodiment, showing operation regions of the user's hand with referenceto the user's trunk axis,

FIG. 10 is a schematic view of the input device according to the thirdembodiment, showing the operation regions of the user's hand withrespect to the longitudinal position at the center point of the user'strunk,

FIG. 11 is a schematic view of the input device according to the thirdembodiment, showing the operation regions with respect to an imaginaryboundary line created by using positional information of the user'sface,

FIG. 12 is a flowchart illustrating the operations of the input deviceaccording to the third embodiment,

FIG. 13 is a schematic view of an input device according to a fourthembodiment, showing regions of the user's gesture operation,

FIGS. 14A, B, and C are schematic views of the input device according tothe fourth embodiment, illustrating the position of the user's hand andcorresponding display screens,

FIG. 15 is a flowchart illustrating the operations of the input deviceaccording to the fourth embodiment, and

FIG. 16 is a schematic view of an input device according to a fifthembodiment, showing the user's gesture.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present invention are described withreference to drawings.

First Embodiment

An input device 100 according to the present embodiment is a devicecapable of detecting the position of a user's body part from a movingimage captured from the user, detecting the user's motion (hereinafterreferred to as “gesture”), and according to the detected motion,executing a predetermined processing and also changing the display ofthe GUI (abbreviation of Graphical User Interface).

FIG. 1 is a schematic view of an operation environment in which the useroperates the input device 100.

An imaging unit 101 is a camera for inputting a moving image to theinput device 100. A display unit 110 is a display provided in the inputdevice and formed by such a display apparatus as, for example, a liquidcrystal display, plasma display, or the like. The display unit 110includes a display panel, a panel control circuit, and a panel controldriver, and displays an image formed by data supplied from an imageprocessing unit 210 which is be described later, to a display panel.

A first menu 111 and a second menu 112 are GUIs displayed on the displayunit 111, each of which is associated with a predetermined useroperation. In this embodiment, the first menu 111 serves as a GUI forthe volume adjustment, and the second menu 112 serves as a GUI for thechannel selection. However, those menus may be associated with otheroperations.

A user 120 is a user who performs an operation with respect to the inputdevice 100. User's operation is performed, for example, when the user'sface 122 is directed toward the input device 100, by a user's gesture ofwaving the user's hand 121 horizontally or vertically, swiping the hand121 upward, downward, rightward, or leftward from a predeterminedposition or circling the hand 121. Then, the input device 100 recognizesthe gesture and executes a predetermined processing according to therecognition result of the gesture.

FIG. 2 is a block diagram showing an example of the configuration of theinput device 100. The input device 100 includes, for example, theimaging unit 101, a system control unit 200, an image processing unit210, and a display unit 110, as shown in FIG. 2. The system control unit200 includes a sub-unit which achieves functions of a position detectionunit 201 and a gesture recognition unit 202.

The position detection unit 201 detects positions of user's body partssuch as the hand, face and trunk from a moving image obtained from theimaging unit 101 by using, for example, such a processing as the shapedetection algorithm or the like. The gesture recognition unit 202 is aunit which recognizes a predetermined gesture operating the input device100 by using positional information of the user's hand detected by theposition detection unit 201. Although this embodiment cites an exampleof recognizing a gesture of the hand, the present invention is notlimited thereto. For example, the gesture recognition unit 202 may beconfigured so as to recognize a gesture of the face or leg or a combinedgesture of multiple body parts.

The system control unit 200 is a unit which performs data processing ofdata such as the output from the position detection unit 201 and theoutput from the gesture recognition unit 202. The system control unit200 may be achieved by CPU execution of a software module stored in thememory or by a dedicated hardware circuit.

The image processing unit 210 includes, for example, a processing unitsuch as ASIC, FPGA and MPU. The image processing unit 210 outputs imagedata of the GUI by converting into a format which can be processed bythe display unit 110, in accordance with control by the system controlunit 200.

When mounted on a TV or a monitor provided with a display device, theinput device 100 enables the operation of such an apparatus by theuser's gesture. Although FIG. 2 shows the input device 100 has aconfiguration including the display unit 100, the input device 100 needsnot to necessarily be provided with the display unit 110. For example,the input device may be mounted on a disk player or the like notprovided with a display unit in such a manner that the image is providedto the user by utilizing an external display unit connected to the diskplayer. Moreover, for example, a video camera provided with an imagingunit may be connected to a display unit, disk player or the like in sucha manner that processing is performed by the display unit, disk playeror the like based on the image captured by the video camera.

Although FIG. 2 shows a configuration used for illustrating thisembodiment and other embodiments, it will be apparent to those skilledin the art that other configurations may be added, if necessary. Forexample, when the input device 100 is mounted on a TV which receives abroadcast, the TV may be provided with a tuner or demultiplexer used forreception and display of the broadcast, a decoder, a speaker whichoutputs the voice, or the like.

FIG. 3 shows a view of the user operating the input device by thegesture according to this embodiment. The position of the user's hand121 is represented, for example, by a coordinate formed by a lateralposition 302 at a center point 320 of a shape region of the detectedhand and a longitudinal position 303 at the center point 320. Theposition of the user's face 122 is represented, for example, by acoordinate formed by a lateral position 300 at a center point 310 of thedetected face and a longitudinal position 301 at the center point 310.

In this embodiment, a method of separating a detection range of theuser's hand 121 to an operation region 304 located to the right of aline passing the lateral position 300 of the user's face and to anoperation region 305 located to the left of the line is described.

Here, depending on the data processing by the system control unit 200,there are two modes in which the position of the hand 121 of the userperforming a gesture is detected; that is, a mode in which the positionof the hand 121 is detected in a region to the right of the lateralposition 300 (operation region 304) of the user's face 122, and a modein which the position of the hand 121 is detected in a region to theleft of the lateral position 300 (operation region 305) of the face 122.In the input device 100 according to this embodiment, a differentoperation is allocated with respect to a predetermined gesture performedby the user depending on the two modes.

FIG. 4 shows schematic views of the display unit 110 when the user'shand 121 using the input device 100 is detected in the operation regions304 and 305 respectively. FIG. 4A shows the user 120 and a mode of thedisplay on the input device 100 when the user's hand is detected in theoperation region 304. The input device 100 shows the user 120 byhighlighting display of the first menu 111, or the like that anoperation associated with the first menu (in this embodiment, volumeadjustment) is executed when the user performs a predetermined gesturein this mode.

FIG. 4B shows the user 120 and a mode of the display screen on the inputdevice 100 when the user's hand is detected in the operation region 305(left to the face). The input device 100 shows the user 120 byhighlighting display of the second menu 112, or the like that anoperation associated with the second menu (in this embodiment, channelselection) is executed when the user performs a predetermined gesture inthis mode.

Hereinafter, the operation of the input device 100 by the gestureaccording to the first embodiment is described using a flowchart shownin FIG. 5. Detection processing of the user's gesture is a processingperformed by the system control unit 200 shown in FIG. 2.

Firstly, the system control unit 200 detects the positions of the user'shand 121 and the face 122, and calculates the position of the hand withrespect to the face (Step 500).

According to the result in the Step 500, the system control unit 200changes display of the GUI depending on the detected position of thehand with respect to the face (Step 501). As shown in FIG. 4, the firstmenu 111 is highlighted when the hand is detected to the right of theface, while the second menu 112 is highlighted when the hand is detectedto the left of the face. This step enables the user to predict whichoperation is executed by the gesture which the user is going to perform.

Although this embodiment is configured so as to highlight a menucorresponding to the detected position of the hand, the display methodis not limited thereto if a display allows the user to recognize whichoperation is executed by the gesture. For example, the display may beconfigured in such a manner that when the hand is detected to the rightof the face, only the first menu 111 is highlighted, but not the secondmenu 112.

When no positional information of the hand and face exist due todetection failure of the hand and face, or the like, the display unit110 displays neither the first menu 111 nor the second menu 112, forexample.

Next, the system control unit 200 recognizes the gesture (Step 502),and, if a pre-registered gesture is not recognized, the system controlunit 200 terminates the processing (Step 503: No). If the pre-registeredgesture is recognized, the system control unit 200 proceeds to Step 504(Step 503: Yes).

Next, the system control unit 200 determines whether the hand is to theright or to the left of the face (Step 504).

The system control unit 200 checks whether the determination result inthe Step 504, that is, the position of hand with respect to the face, isconsistent with the position of the GUI highlighted in the above Step501 (Steps 505 and 506). If not consistent, the system control unit 200terminates the processing without accepting the operation by the gesture(Step 505: No, and Step 506: No). If consistent, the system control unit200 proceeds to Step 507 or Step 508 depending on the result of the Step504 (Step 505: Yes, and Step 506: Yes).

Although the first menu 111 (a menu on the right side) is highlighted onthe display unit 110, if the position of the hand is determined to be tothe left of the face when the gesture is recognized according to theresult in the Step 504 (Step 504: Left), the system control unit 200assumes that a gesture recognition error occurs due to the inconsistencybetween the GUI display and an operation to be executed by the inputdevice, and thus accepts no user's gestures.

When a gesture is recognized, if the position of the hand is determinedto be to the right of the face, that is, if the position of the hand isdetected in the operation region 304 shown in FIG. 3 (Step 504: Right),and the determination result thereof is consistent with a display of theGUI (Step 505: Yes), the input device 100 executes an operationassociated with the first menu (in this embodiment, volume adjustment)(Step: 507).

Similarly, When a gesture is recognized, if the position of the hand isdetermined to be to the left of the face, that is, if the position ofthe hand is detected in the operation region 305 shown in FIG. 3 (Step504: Left), and the determination result thereof is consistent with adisplay of the GUI (Step 506: Yes), the input device 100 executes anoperation associated with the second menu (in this embodiment, channelselection) (Step: 508).

Thus, by recognizing the position of the hand with respect to the face,the user's predetermined gesture can be allocated to multipleoperations, and thereby, the user can perform multiple operationswithout storing multiple gestures. Furthermore, since display of thefirst menu 111 is highlighted when the position of the hand is detectedto the right of the face, and display of the second menu 112 ishighlighted when the position of the hand is detected to the left of theface, the user can predict which operation applies to a gesture the useris going to perform. Furthermore, this also provides an effect to reducethe time for menu selection while operating the units, and therebyenables the user to perform a desired operation faster.

Furthermore, compared with the prior art which can be operated by thegesture, the processing load of operations by the gesture can be reducedas well.

In the embodiment described, a method of allocating the user's gestureto two types of operations using the two types of modes is describedusing the two modes, the two types of modes including a mode in whichthe position of the hand is to the right of the face, and a mode inwhich the position of the hand is to the left of the face. However, amode in which the hand is above or below the face also may be used.Moreover, the user's gesture may be allocated to at least three types ofoperations by using such modes in which the position of the hand is tothe upper left, the upper right, the lower right or the lower left ofthe face.

Second Embodiment

Hereinafter, a second embodiment is described with reference to FIGS. 6to 8.

The input device 100 according to the first embodiment displays either acorresponding first or second menu by highlighting when the position ofthe hand is detected to the right or to the left of the face. In thepresent embodiment, the GUI displays the position where the user's handis detected with respect to the face, in addition to the operationmethod according to the first embodiment.

FIG. 6 is a schematic view showing the user operating an input device100. A display unit 110 of the input device 100 is provided with anoperation region display 600 indicating the region in which the user'shand is detected, and a GUI of a hand position indicator 601 indicatingthe position in which the user's hand is detected, in addition to thefirst menu 111 and the second menu 112 according to the firstembodiment. The input device 100 displays the position where the user'shand is detected with respect to the face, using the operation regiondisplay 600 and the hand position indicator 601 in combination. A medianline 602 is an intermediate boundary line separating the operationregion display 600 to right and left portions. The user performs thegesture to operate the input device 100 while checking a screen on thedisplay unit 110 of the input device 100.

FIG. 7 shows a relation between the user and a screen displayed on thedisplay unit 110 according to this embodiment. An operation region range700 is a range in which the gesture can be recognized. According to thisembodiment, a midpoint of the operation region range 700 corresponds toa face's lateral position 300. The lateral length of the operationregion display 600 displayed on the display unit 110 represents theoperation region range 700, and a median line 602 corresponds to theface's lateral position 300.

FIG. 8 is a schematic view of the input device 100 according to thisembodiment, illustrating the position of the user's hand and acorresponding display screen. The position of the hand in the operationregion 700 shown in FIG. 7 corresponds to the hand position indicator601 in the operation region display 600. That is, when the user's hand121 moves rightward in the operation region range 700, the position ofthe hand position indicator 601 is moved to the right of the operationregion display 600 as shown in FIG. 8A. Similarly, when the user's hand121 moves leftward in the operation region range 700, the position ofthe hand position indicator 601 is moved to the left of the operationregion display 600 as shown in FIG. 8B.

With the above configuration, there is an advantageous effect that theuser can identify the position of the user's hand by observing theposition of the hand position indicator 601 in the operation regiondisplay 600.

Third Embodiment

Hereinafter, a third embodiment is described with reference to FIGS. 9to 12.

The input device 100 according to the first and second embodiments isconfigured so as to set a predetermined reference position by usingpositional information of the face and allocate a different operationaccording to the relative position of the hand with respect to thereference position. In this embodiment, a method of allocatingoperations by the gesture by using a relative position of the user'shand with respect to other body parts is described.

Firstly, an imaginary boundary line extending longitudinally orlaterally, or at a predetermined inclination is created with the user'sbody parts as a reference. The created imaginary boundary line producesat least two operation regions. In this embodiment, a method ofallocating an operation to a predetermined gesture depending on theoperation region in which the user's hand is detected is described.Hereinafter, an example of creating operation regions using theimaginary boundary line is described.

FIG. 9 is a schematic view showing operation regions of the user's handwith respect to the user's trunk axis according to this embodiment. Theinput device 100 detects a center point 900 of the user's trunk and theinclination of the trunk by using, for example, the position detectionunit 201. A boundary line 901 passes through the detected center point900 of the trunk, at an inclination thereof same as the inclination ofthe trunk.

The boundary line 901 thus created produces two operation regions forthe hand; an operation region 902 and an operation region 903. Then, twotypes of operations are allocated to the operation of the input device100 by the user's gesture; that is, an operation when the position ofthe hand is detected in the operation region 902, and an operation whenthe position of the hand is detected in the operation region 903.

Thus, according to the user's position, multiple operations can beallocated to the operation of the input device 100 by a gesture moreappropriately and thereby user's usability can be improved. Theimaginary line 901 may be created by other methods such as, for example,a method of defining the imaginary line using a line passing a centerpoint 310 of the face and the center point 900 of the trunk.

FIG. 10 is a schematic view showing operation regions of the operationby the user's gesture with respect to the longitudinal position of thecenter point of the user's trunk according to this embodiment. When animaginary boundary line 901-b extending laterally is created at thelongitudinal position of the center point 900 of the trunk, operationregions 1000 and 1001 shown in FIG. 10 are produced. Then, two types ofoperations are allocated to the operation of the input device 100 by theuser's gesture; i.e. an operation when the position of the hand isdetected in the operation region 1000, and an operation when theposition of the hand is detected in the operation region 1001.

Thus, multiple operations can be allocated to the input device by agesture when the hand is above the center point of the body and by agesture when the hand is below the center point.

FIG. 11 is a schematic view showing the operation regions with respectto a boundary line created by using positional information of the user'sface. It is considered that when operating the input device 100 by thegesture of the right hand, using a line passing through the centerpoints of the face and trunk as the imaginary boundary line makes anoperation allocated to the left of the line makes difficult comparedwith an operation allocated to the right side thereof. Therefore, animaginary boundary line 901-c extending longitudinally is created at aposition apart by a distance 1100 from the lateral position 300 of theuser's face, and thereby operation regions 1101 and 1102 shown in FIG.11 are produced. Two types of operations are allocated to the operationof the input device 100 by the user's gesture: i.e. an operation whenthe position of the hand is detected in the operation region 1101, andan operation when the position of the hand is detected in the operationregion 1102.

By setting the imaginary boundary line 901-c to the right (or to theleft when a gesture is performed with the left hand) of the longitudinalposition of the face in such a manner, an operation by the user's righthand allocated to the left region becomes easier compared with animaginary boundary line passing the center points of the face and trunk.

Furthermore, when the input device 100 determines by using, for example,the position detection unit 202 configured to determine whether thegesture is performed with a right hand or left hand, that the userperforms the gesture with the left hand, the gesture of the left handcan be performed easily by setting the boundary line 901-c to the leftof the face.

Hereinafter, a method of allocating two types of operations to the inputdevice 100 with respect to a predetermined gesture according to thisembodiment is described.

Detection processing of the input operation is a processing performed bythe system control unit 200 shown in FIG. 2. Firstly, the system controlunit 200 detects the position of the user's hand 121 and the position ofa body part serving as a reference for creating operation regions (Step1200).

Using positional information of the user's body part being the output inthe Step 1200, the system control unit 200 creates, for example, animaginary boundary line such as shown in FIGS. 9 to 11 to produce twooperation regions (Step 1201).

Next, similarly with the Step 501 in the flowchart of FIG. 5, the systemcontrol unit 200 changes the GUI display depending on the operationregion in which the hand is detected (Step 1202).

Next, similarly with the Step 502 in the flowchart of FIG. 5, the systemcontrol unit 200 recognizes the gesture (Step 1203). If a pre-registeredgesture is not recognized, the system control unit 200 terminates theprocessing (Step 1204: No). If the pre-registered gesture is recognized,the system control unit 200 proceeds to Step 1205 (Step 1204: Yes).

Next, the system control unit 200 determines whether the hand isdetected in the first operation region or the second operation region(Step 1205).

Next, the system control unit 200 checks whether or not thedetermination result in the Step 1205 is consistent with the GUIprocessed in the Step 1202 (Steps 1206 and 1207). If not consistent, thesystem control unit 200 terminates the processing without accepting theoperation by the gesture (Step 1206: No, and Step 1207: No). Ifconsistent, the system control unit 200 proceeds to Step 1208 or Step1209 depending on the result of the Step 1205 (Step 1206: Yes, and Step1207: Yes).

For example, although the first menu 111 (a menu on the right side) ishighlighted on the display unit 110 in the Step 1202, if, when thegesture is recognized, the position of the hand is determined to be tothe left of the face according to the result in the Step 1205, thesystem control unit 200 assumes that a gesture recognition error occursdue to the inconsistency between the GUI display observed by the userand an operation to be executed by the input device, and accepts nooperations by the user's gesture.

If, when a gesture is performed, the position of the hand is in thefirst operation region (Step 1205: first operation region) and thedetermination result thereof is consistent with a displayed GUI (Step1206: Yes), the input device 100 executes a first user operation (Step:1208). Similarly, if, when a gesture is performed, the position of thehand is in the second operation region (Step 1205: second operationregion) and the determination result thereof is consistent with adisplayed GUI (Step 1207: Yes), the input device 100 executes a seconduser operation (Step 1209).

In such a manner, the imaginary boundary line and the operation regionsare defined using positional information of user's body parts. Bysetting two modes of operations according to the operation region wherethe position of the hand is detected, two types of user's operations canbe allocated to the operation of the input device 100 when apredetermined gesture is performed. The imaginary boundary line may beconfigured in such a manner as to be set appropriately according to amethod of separating into operation regions where the position of theuser's hand is recognized. For example, if the operation region wherethe position of the user's hand is recognized is desired to be separatedinto left and right parts, the imaginary boundary line may be definedwith a line passing through the center point 310 of the user's face andthe center point 900 of the user's trunk. Furthermore, if the operationregion where the position of the user's hand is recognized is desired tobe separated into upper and lower parts, the imaginary boundary line maybe defined with a line linking both ends of the user's shoulder.Moreover, at least two imaginary boundary lines can be defined to createat least three regions where the position of the user's hand isrecognized. In such a manner, at least three types of operations can beallocated to a predetermined gesture.

Fourth Embodiment

Hereinafter, a fourth embodiment is described with reference to FIGS. 13to 15.

In the first, second and third embodiments, methods for allocatingmultiple operations to the input device 100 by using such modes in whichthe position of the hand is to the right or left of, or below or above apredetermined reference position are described. In this embodiment, amethod for allocating multiple operations to the input device 100 byusing a distance of the hand from a predetermined reference position isdescribed.

FIG. 13 shows the user operating the input device 100 according to thisembodiment. An operation boundary line 901-d is an imaginary operationboundary line which is set, for example, from the positions ofpredetermined body parts (face, shoulder, or the like). Here, imaginaryboundary lines (a boundary line 1300 on the right side, and a boundaryline 1301 on the left side, as viewed from the user) are set each onboth right and left sides of the operation boundary line 901-d such thata first operation region 1302, a second operation region 1303 and athird operation region 1303 are produced, the imaginary boundary linesbeing each apart rightward and leftward by a predetermined distance fromthe operation boundary line 901-d. Then, three conditions are setaccording to the operation region in which the position of the hand isdetected when the user operates the input device by the gesture; thethree conditions including: a case where the position of the hand isdetected in the first operation region 1302, a case where the positionof the hand is detected in the second operation region 1303, and a casewhere the position of the hand is detected in the third operation region1304. Then, using the three conditions, a different operation isallocated to the input device 100. An operation region range 700-b issimilar with the operation region range 700 shown in FIG. 7.

FIG. 14 shows the positions of the user's hand and corresponding displayscreens on the display unit 110 of the input device 100 when the user'shand is detected in the above three operation regions.

FIG. 14A shows a relation between the user and a screen on the displayunit 110, when the user's hand is detected in the first operation region1302. The display unit 110 displays a first menu 111 and a second menu112. To indicate that both of the menus are not operable, for example,display intensity of those menus may be reduced. A hand positionindicator 601 in an operation region display 600 is displayed in thevicinity of the middle position thereof. In this embodiment, nooperation is allocated to the region, so that the user cannot operatethe input device 100 even when the gesture is performed.

FIG. 14B shows relation between the user and a screen on the displayunit 110 when the user's hand is detected in the second operation region1303. On the display unit 110, the first menu 111 is highlighted moreintensively than the second menu 112, showing the user that an operationshown by the first menu 111 is available by performing a predeterminedgesture. The operation region display 600 displays the hand positionindicator 601 on the right side. Then, when the user performs thegesture, the first operation (in this embodiment, volume adjustment) isexecuted to the input device 100.

FIG. 14C shows a relation between the user and a screen on the displayunit 110 when the user's hand is detected in the third operation region1304. On the display unit 110, the second menu 112 is highlighted moreintensively than the first menu 111, showing the user that an operationshown by the second menu 112 is available by performing a predeterminedgesture. The operation region display 600 displays the hand positionindicator 601 on the left side. Then, when the user performs thegesture, the second operation (in this embodiment, channel selection) isexecuted to the input device 100.

Hereinafter, a method for allocating a predetermined gesture operationaccording to this embodiment to multiple operations is described withreference to a flowchart shown in FIG. 15.

Firstly, the system control unit 200 detects the positions of the user'shand 121 and a body part serving as a reference for creating operationregions (Step 1500). FIG. 13 shows that the user's shoulder serves as areference for setting the operation regions.

Using positional information of the user's body part which is the outputin the Step 1500, the system control unit 200 determines an imaginaryboundary line, for example, as shown in FIG. 13 and defines three typesof operation regions, each of which being apart by a predetermineddistance from the imaginary boundary line (Step 1501).

Then, the system control unit 200 changes the GUI display according tothe operation region where the hand is detected (Step 1502). As shown inFIG. 14, the GUI display is changed according to the operation regionwhere the user's hand is detected.

Although this embodiment is configured such that a menu associated withthe detected position of the hand is highlighted, the display method isnot limited thereto so far as a display is configured in such a mannerthat the user can recognize the type of an operation which will beexecuted by the gesture. For example, it may be configured in such a waythat when the hand is detected in the operation region 1303, only thefirst menu 111 is displayed, but not the second menu 112, and when thehand is detected in the operation region 1302, no menu is displayed.Furthermore, it may be configured in such a manner that the operationregion display 600 and the hand position indicator 601 are notdisplayed.

Next, the system control unit 200 performs the gesture recognitionprocessing (Step 1503). If a pre-registered gesture is not recognized,the system control unit 200 terminates the processing (Step 1504: No).If the pre-registered gesture is recognized, the system control unit 200proceeds to Step 1505 (Step 1504: Yes).

Next, the system control unit 200 determines whether the hand isdetected in the first operation region, the second operation region orthe third operation region (Step: 1505).

Next the system control unit 200 checks whether or not the determinationresult in the Step 1505 is consistent with the GUI displayed accordingto the processing in the Step 1502 (Step 1506, Step 1507, and Step1508). If not consistent, the system control unit 200 terminates theprocessing without accepting the operation by the gesture (Step 1506:No, Step 1507: No, and Step 1508: No). If consistent, the system controlunit 200 proceeds to Step 1509, Step 1510 or Step 1511 depending on theresult in the Step 1505.

For example, although the display unit 110 highlights the first menu 111(a menu on the right side) in the Step 1502, if the detected position ofthe user's hand determined in the Step 1505 is the left region 1304, thesystem control unit 200 accepts no user's operations by the gesture.

When the gesture is performed, if it is determined that the user's handis detected in the first region (Step 1505: first operation region) andthe determination result thereof is consistent with a GUI displayed as aresult of processing in the Step 1502 (Step 1506: Yes), the input device100 executes the first user operation (Step 1509).

In a similar manner, if it is determined that the user's hand isdetected in the second region or in the third region when the gesture isperformed, the input device 100 executes a corresponding user operation(Step 1507, Step 1510, Step 1508, Step 1511).

Thus, by defining multiple operation regions depending on a distancefrom a boundary line which is set based on the positions of the user'spredetermined body parts, multiple operations can be allocated to theinput device 100 according to the position where the user's hand isdetected.

Although this embodiment cites three operation regions as an example,the number of operation regions may be reduced or increased by reducingor increasing the number of boundary lines. Furthermore, in thisembodiment, a boundary line is set based on predetermined body parts,and multiple operation regions are set depending on a distance from theboundary line. However, it may be configured such that multipleoperation regions are set depending on a distance from a predeterminedbody part without setting a boundary line.

Fifth Embodiment

Hereinafter, a fifth embodiment is described with reference to FIG. 16.

In this embodiment, a method of defining user's gestures operating theinput device 100 using the positions of predetermined body parts.

In prior arts, a swiping gesture is defined when the hand continuouslymoves rightward or leftward from a position to another position within acamera's image capturing range over a specific time period. However,since information of the camera's image capturing range is used, a sameswiping gesture performed by a user may not be recognized in some casesdue to the difference in the user's body height or depending on thedistance from an input device to a place where the user operates theinput device.

In this embodiment, the swiping gesture is defined with the position ofa user's body part actually captured, instead of the camera's imagecapturing range.

FIG. 16 is a schematic view showing a gesture performed by the user withrespect to an input device 100 according to this embodiment. Forexample, a gesture 1600 is defined to be formed when the user's handmoves from the operation region 1302 to the operation region 1304. Sincethe position of a boundary line 1201 is set based on positionalinformation of detected user's body parts, the size of the operationregions 1302 and 1304 vary according to the size and position of theuser's body. Thus, the input device can be operated appropriately evenby multiple users of different body heights or body builds or by a sameuser operating from different positions.

By defining the gesture by using positional information of body parts insuch a manner, operation of the input device 100 by a gesture capable ofprocessing a change of the user or the operation position can beachieved.

Positions of body parts such as center points of the face and trunkdetected by the input device 100 may be used as a reference for definingother gestures (such as hand waving, hand circling, or the like). Forexample, the input device may be configured in such a manner as todetermine as the gesture of circling the hand when the user's hand ismoving around the center point of the trunk, when the circling radius ofthe user's hand is within a predetermined range calculated on the basisof the lateral width of the trunk, or the like.

Although the above first to fifth embodiments are described onassumption of a device which includes the display unit 110 configured todisplay an image and the video processing unit 210, the presentinvention is not limited thereto. If a device includes a cameraconfigured to capture the image and a control unit configured to controlthe device on the basis of the image captured by the camera, operationsof the device by the gesture described in the above first to fifthembodiments are possible based on the user's image captured by thecamera. In this case, menus cannot be highlighted, but multipleoperations can be allocated to a same gesture. Moreover, the device maybe configured in such a manner that an operation corresponding to thegesture is shown to the user by light emitting elements (by utilizingthe position or light color of the light emitting elements, or the like)such as LED (Light Emitting Diode) provided in the device, instead ofthe menu highlighting. The input device also may be configured in such amanner that the processing described in the above embodiments isexecuted by a software application.

A device including an imaging element such as a camera can providefunctions and effects similar with those described in the aboveembodiments by executing software capable of performing the processingdescribed in the above embodiments by using CPU of the device.

While we have shown and described several embodiments in accordance withour invention, it should be understood that disclosed embodiments aresusceptible of changes and modifications without departing from thescope of the invention. Therefore, we do not intend to be bound by thedetails shown and described herein but intend to cover all such changesand modifications that fall within the ambit of the appended claims.

1. An image processing device which outputs an image signal, the imageprocessing device including: a camera configured to capture the image,and a control unit configured to control the image processing devicebased on the image captured by the camera, wherein the control unitdetects multiple body parts of the user from the image captured by thecamera and selects a type of the operation with respect to the imageprocessing device based on the detected positions of the multiple bodyparts, and wherein when a user's predetermined gesture is recognizedfrom the image captured by the camera, the image processing unitexecutes the operation of the selected type.
 2. The image processingdevice according to claim 1, further including an image processing unitconfigured to process the image signal to be output, wherein the imageprocessing unit processes the image signal in such a manner that a menushowing an operation selected by the control unit is contained in theoutput image signal.
 3. The image processing device according to claim1, wherein the control unit detects positions of the user's face andhand from the image captured by the camera, and wherein when a detectedposition of the hand is to the right of a detected position of the face,the control unit selects a first operation, and when the detectedposition of the hand is to the left of the detected position of theface, the control unit selects a second operation.
 4. The imageprocessing device according to claim 1, wherein the control unit detectspositions of user's hand and predetermined body part other than the handfrom the image captured by the camera, wherein when a detected positionof the hand is in a first region separated by a boundary line createdbased on the detected predetermined body, the control unit selects thefirst operation, and when the detected position of the hand is in asecond region separated by the boundary line created based on thedetected predetermined body, the control unit selects the secondoperation.
 5. The image processing device according to claim 1, whereinthe control unit detects positions of the user's hand and predeterminedbody part other than the hand from the image captured by the camera, andselects a type of the operation with respect to the image processingdevice based on a distance between the detected positions of the faceand predetermined body part.
 6. The image processing device according toclaim 3, wherein an image signal being output contains an image signalassociated with an operation region display in which the position of theuser's hand detected by the control unit is shown.
 7. The imageprocessing device according to claim 4, wherein an image signal beingoutput contains an image signal associated with an operation regiondisplay in which the position of the user's hand detected by the controlunit is shown.
 8. The image processing device according to claim 5,wherein an image signal being output contains an image signal associatedwith an operation region display in which the position of the user'shand detected by the control unit is shown.
 9. An image processingdevice which outputs the image signal, the image processing deviceincluding: a camera configured to capture the image, and a control unitconfigured to execute a processing associated with a user'spredetermined gesture when a user's gesture contained in the imagecaptured by the camera is the predetermined gesture, wherein the controlunit detects positions of the user's hand and predetermined body partother than the hand from the image captured by the camera, and when thedetected position of the hand moves from a first region to a secondregion, the control unit determines that the user's gesture is thepredetermined gesture, the first and second regions being set based onthe position of the predetermined body part other than the detectedhand.
 10. An image display device which displays an image, the imagedisplay device including: a camera configured to capture the image, acontrol unit configured to control the image display device based on theimage captured by the camera, an image processing unit configured toprocess the image to be displayed, and a display unit configured todisplay the image processed by the image processing unit, wherein whenthe camera captures an image in which the user of the image displaydevice performs a predetermined gesture with the user's hand to theright of the user's face, the control unit performs a first control,whereas when the camera captures the image in which the user of theimage display device performs a predetermined gesture with the user'shand to the left of the user's face, the control unit performs a secondcontrol.
 11. An image display device which displays the image, the videodisplay device including: a camera configured to capture the image, acontrol unit configured to control the image display device based on theimage captured by the camera, an image processing unit configured toprocess the image to be displayed, and a display unit configured todisplay the image processed by the image processing unit, wherein whenthe camera captures the image in which the user of the image displaydevice performs a predetermined gesture with the user's hand to theright of the user's shoulder, the control unit performs a first control,whereas when the camera captures the image in which the user of theimage display device performs a predetermined gesture with the user'shand to the left of the user's shoulder, the control unit performs asecond control.
 12. The image display device according to claim 10,further including a speaker configured to generate the voice, whereinthe first control or the second control is a control associated with thevolume of voice to be output from the speaker.
 13. The image displaydevice according to claim 11, further including a speaker configured togenerate the voice, wherein the first control or the second control is acontrol associated with the volume of voice to be output from thespeaker.
 14. The image display device according to claim 10, furtherincluding a tuner for broadcasting, wherein the first control or thesecond control is a control associated with the channel selection ofbroadcast waves by the tuner.
 15. The image display device according toclaim 11, further including a tuner for broadcasting, wherein the firstcontrol or the second control is a control associated with the channelselection of broadcast waves by the tuner.